{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# How to stream from a question-answering chain\n",
        "\n",
        ":::info Prerequisites\n",
        "\n",
        "This guide assumes familiarity with the following:\n",
        "\n",
        "- [Retrieval-augmented generation](/docs/tutorials/rag/)\n",
        "\n",
        ":::\n",
        "\n",
        "Often in Q&A applications it's important to show users the sources that were used to generate the answer. The simplest way to do this is for the chain to return the Documents that were retrieved in each generation.\n",
        "\n",
        "We'll be using the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng for retrieval content this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n",
        "### Dependencies\n",
        "\n",
        "We’ll use an OpenAI chat model and embeddings and a Memory vector store in this walkthrough, but everything shown here works with any [ChatModel](/docs/concepts/chat_models) or [LLM](/docs/concepts/text_llms), [Embeddings](/docs/concepts/embedding_models), and [VectorStore](/docs/concepts/vectorstores) or [Retriever](/docs/concepts/retrievers).\n",
        "\n",
        "We’ll use the following packages:\n",
        "\n",
        "```bash\n",
        "npm install --save langchain @langchain/openai cheerio\n",
        "```\n",
        "\n",
        "We need to set environment variable `OPENAI_API_KEY`:\n",
        "\n",
        "```bash\n",
        "export OPENAI_API_KEY=YOUR_KEY\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### LangSmith\n",
        "\n",
        "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with [LangSmith](https://smith.langchain.com/).\n",
        "\n",
        "Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
        "\n",
        "\n",
        "```bash\n",
        "export LANGSMITH_TRACING=true\n",
        "export LANGSMITH_API_KEY=YOUR_KEY\n",
        "\n",
        "# Reduce tracing latency if you are not in a serverless environment\n",
        "# export LANGCHAIN_CALLBACKS_BACKGROUND=true\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Chain with sources\n",
        "\n",
        "Here is Q&A app with sources we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [Returning sources](/docs/how_to/qa_sources/) guide:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{\n",
              "  question: \u001b[32m\"What is Task Decomposition\"\u001b[39m,\n",
              "  context: [\n",
              "    Document {\n",
              "      pageContent: \u001b[32m\"Fig. 1. Overview of a LLM-powered autonomous agent system.\\n\"\u001b[39m +\n",
              "        \u001b[32m\"Component One: Planning#\\n\"\u001b[39m +\n",
              "        \u001b[32m\"A complicated ta\"\u001b[39m... 898 more characters,\n",
              "      metadata: {\n",
              "        source: \u001b[32m\"https://lilianweng.github.io/posts/2023-06-23-agent/\"\u001b[39m,\n",
              "        loc: { lines: \u001b[36m[Object]\u001b[39m }\n",
              "      }\n",
              "    },\n",
              "    Document {\n",
              "      pageContent: \u001b[32m'Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are'\u001b[39m... 887 more characters,\n",
              "      metadata: {\n",
              "        source: \u001b[32m\"https://lilianweng.github.io/posts/2023-06-23-agent/\"\u001b[39m,\n",
              "        loc: { lines: \u001b[36m[Object]\u001b[39m }\n",
              "      }\n",
              "    },\n",
              "    Document {\n",
              "      pageContent: \u001b[32m\"Agent System Overview\\n\"\u001b[39m +\n",
              "        \u001b[32m\"                \\n\"\u001b[39m +\n",
              "        \u001b[32m\"                    Component One: Planning\\n\"\u001b[39m +\n",
              "        \u001b[32m\"                 \"\u001b[39m... 850 more characters,\n",
              "      metadata: {\n",
              "        source: \u001b[32m\"https://lilianweng.github.io/posts/2023-06-23-agent/\"\u001b[39m,\n",
              "        loc: { lines: \u001b[36m[Object]\u001b[39m }\n",
              "      }\n",
              "    },\n",
              "    Document {\n",
              "      pageContent: \u001b[32m\"Resources:\\n\"\u001b[39m +\n",
              "        \u001b[32m\"1. Internet access for searches and information gathering.\\n\"\u001b[39m +\n",
              "        \u001b[32m\"2. Long Term memory management\"\u001b[39m... 456 more characters,\n",
              "      metadata: {\n",
              "        source: \u001b[32m\"https://lilianweng.github.io/posts/2023-06-23-agent/\"\u001b[39m,\n",
              "        loc: { lines: \u001b[36m[Object]\u001b[39m }\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  answer: \u001b[32m\"Task decomposition is a technique used to break down complex tasks into smaller and simpler steps fo\"\u001b[39m... 230 more characters\n",
              "}"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import \"cheerio\";\n",
        "import { CheerioWebBaseLoader } from \"@langchain/community/document_loaders/web/cheerio\";\n",
        "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
        "import { MemoryVectorStore } from \"langchain/vectorstores/memory\"\n",
        "import { OpenAIEmbeddings, ChatOpenAI } from \"@langchain/openai\";\n",
        "import { pull } from \"langchain/hub\";\n",
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
        "import { formatDocumentsAsString } from \"langchain/util/document\";\n",
        "import { RunnableSequence, RunnablePassthrough, RunnableMap } from \"@langchain/core/runnables\";\n",
        "import { StringOutputParser } from \"@langchain/core/output_parsers\";\n",
        "\n",
        "const loader = new CheerioWebBaseLoader(\n",
        "  \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
        ");\n",
        "\n",
        "const docs = await loader.load();\n",
        "\n",
        "const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });\n",
        "const splits = await textSplitter.splitDocuments(docs);\n",
        "const vectorStore = await MemoryVectorStore.fromDocuments(splits, new OpenAIEmbeddings());\n",
        "\n",
        "// Retrieve and generate using the relevant snippets of the blog.\n",
        "const retriever = vectorStore.asRetriever();\n",
        "const prompt = await pull<ChatPromptTemplate>(\"rlm/rag-prompt\");\n",
        "const llm = new ChatOpenAI({ model: \"gpt-3.5-turbo\", temperature: 0 });\n",
        "\n",
        "const ragChainFromDocs = RunnableSequence.from([\n",
        "  RunnablePassthrough.assign({ context: (input) => formatDocumentsAsString(input.context) }),\n",
        "  prompt,\n",
        "  llm,\n",
        "  new StringOutputParser()\n",
        "]);\n",
        "\n",
        "let ragChainWithSource = new RunnableMap({ steps: { context: retriever, question: new RunnablePassthrough() }})\n",
        "ragChainWithSource = ragChainWithSource.assign({ answer: ragChainFromDocs });\n",
        "\n",
        "await ragChainWithSource.invoke(\"What is Task Decomposition\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see what this prompt actually looks like. You can also view it [in the LangChain prompt hub](https://smith.langchain.com/hub/rlm/rag-prompt):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\n",
            "Question: {question} \n",
            "Context: {context} \n",
            "Answer:\n"
          ]
        }
      ],
      "source": [
        "console.log(prompt.promptMessages.map((msg) => msg.prompt.template).join(\"\\n\"));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Streaming final outputs\n",
        "\n",
        "With [LCEL](/docs/concepts/lcel), we can stream outputs as they are generated:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{ question: \"What is task decomposition?\" }\n",
            "{\n",
            "  context: [\n",
            "    Document {\n",
            "      pageContent: \"Fig. 1. Overview of a LLM-powered autonomous agent system.\\n\" +\n",
            "        \"Component One: Planning#\\n\" +\n",
            "        \"A complicated ta\"... 898 more characters,\n",
            "      metadata: {\n",
            "        source: \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
            "        loc: { lines: [Object] }\n",
            "      }\n",
            "    },\n",
            "    Document {\n",
            "      pageContent: 'Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are'... 887 more characters,\n",
            "      metadata: {\n",
            "        source: \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
            "        loc: { lines: [Object] }\n",
            "      }\n",
            "    },\n",
            "    Document {\n",
            "      pageContent: \"Agent System Overview\\n\" +\n",
            "        \"                \\n\" +\n",
            "        \"                    Component One: Planning\\n\" +\n",
            "        \"                 \"... 850 more characters,\n",
            "      metadata: {\n",
            "        source: \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
            "        loc: { lines: [Object] }\n",
            "      }\n",
            "    },\n",
            "    Document {\n",
            "      pageContent: \"(3) Task execution: Expert models execute on the specific tasks and log results.\\n\" +\n",
            "        \"Instruction:\\n\" +\n",
            "        \"\\n\" +\n",
            "        \"With \"... 539 more characters,\n",
            "      metadata: {\n",
            "        source: \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
            "        loc: { lines: [Object] }\n",
            "      }\n",
            "    }\n",
            "  ]\n",
            "}\n",
            "{ answer: \"\" }\n",
            "{ answer: \"Task\" }\n",
            "{ answer: \" decomposition\" }\n",
            "{ answer: \" is\" }\n",
            "{ answer: \" a\" }\n",
            "{ answer: \" technique\" }\n",
            "{ answer: \" used\" }\n",
            "{ answer: \" to\" }\n",
            "{ answer: \" break\" }\n",
            "{ answer: \" down\" }\n",
            "{ answer: \" complex\" }\n",
            "{ answer: \" tasks\" }\n",
            "{ answer: \" into\" }\n",
            "{ answer: \" smaller\" }\n",
            "{ answer: \" and\" }\n",
            "{ answer: \" simpler\" }\n",
            "{ answer: \" steps\" }\n",
            "{ answer: \".\" }\n",
            "{ answer: \" It\" }\n",
            "{ answer: \" can\" }\n",
            "{ answer: \" be\" }\n",
            "{ answer: \" done\" }\n",
            "{ answer: \" through\" }\n",
            "{ answer: \" various\" }\n",
            "{ answer: \" methods\" }\n",
            "{ answer: \" such\" }\n",
            "{ answer: \" as\" }\n",
            "{ answer: \" using\" }\n",
            "{ answer: \" prompting\" }\n",
            "{ answer: \" techniques\" }\n",
            "{ answer: \",\" }\n",
            "{ answer: \" task\" }\n",
            "{ answer: \"-specific\" }\n",
            "{ answer: \" instructions\" }\n",
            "{ answer: \",\" }\n",
            "{ answer: \" or\" }\n",
            "{ answer: \" human\" }\n",
            "{ answer: \" inputs\" }\n",
            "{ answer: \".\" }\n",
            "{ answer: \" Another\" }\n",
            "{ answer: \" approach\" }\n",
            "{ answer: \" involves\" }\n",
            "{ answer: \" outsourcing\" }\n",
            "{ answer: \" the\" }\n",
            "{ answer: \" planning\" }\n",
            "{ answer: \" step\" }\n",
            "{ answer: \" to\" }\n",
            "{ answer: \" an\" }\n",
            "{ answer: \" external\" }\n",
            "{ answer: \" classical\" }\n",
            "{ answer: \" planner\" }\n",
            "{ answer: \".\" }\n",
            "{ answer: \"\" }\n"
          ]
        }
      ],
      "source": [
        "for await (const chunk of await ragChainWithSource.stream(\"What is task decomposition?\")) {\n",
        "  console.log(chunk)\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can add some logic to compile our stream as it's being returned:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "question: \"What is task decomposition?\"\n",
            "\n",
            "\n",
            "context: [{\"pageContent\":\"Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\",\"metadata\":{\"source\":\"https://lilianweng.github.io/posts/2023-06-23-agent/\",\"loc\":{\"lines\":{\"from\":176,\"to\":181}}}},{\"pageContent\":\"Task decomposition can be done (1) by LLM with simple prompting like \\\"Steps for XYZ.\\\\n1.\\\", \\\"What are the subgoals for achieving XYZ?\\\", (2) by using task-specific instructions; e.g. \\\"Write a story outline.\\\" for writing a novel, or (3) with human inputs.\\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\\nSelf-Reflection#\",\"metadata\":{\"source\":\"https://lilianweng.github.io/posts/2023-06-23-agent/\",\"loc\":{\"lines\":{\"from\":182,\"to\":184}}}},{\"pageContent\":\"Agent System Overview\\n                \\n                    Component One: Planning\\n                        \\n                \\n                    Task Decomposition\\n                \\n                    Self-Reflection\\n                \\n                \\n                    Component Two: Memory\\n                        \\n                \\n                    Types of Memory\\n                \\n                    Maximum Inner Product Search (MIPS)\\n                \\n                \\n                    Component Three: Tool Use\\n                \\n                    Case Studies\\n                        \\n                \\n                    Scientific Discovery Agent\\n                \\n                    Generative Agents Simulation\\n                \\n                    Proof-of-Concept Examples\\n                \\n                \\n                    Challenges\\n                \\n                    Citation\\n                \\n                    References\",\"metadata\":{\"source\":\"https://lilianweng.github.io/posts/2023-06-23-agent/\",\"loc\":{\"lines\":{\"from\":112,\"to\":146}}}},{\"pageContent\":\"(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\",\"metadata\":{\"source\":\"https://lilianweng.github.io/posts/2023-06-23-agent/\",\"loc\":{\"lines\":{\"from\":277,\"to\":280}}}}]\n",
            "\n",
            "\n",
            "answer: \"\"\n",
            "Task\n",
            " decomposition\n",
            " is\n",
            " a\n",
            " technique\n",
            " used\n",
            " to\n",
            " break\n",
            " down\n",
            " complex\n",
            " tasks\n",
            " into\n",
            " smaller\n",
            " and\n",
            " simpler\n",
            " steps\n",
            ".\n",
            " It\n",
            " can\n",
            " be\n",
            " done\n",
            " through\n",
            " various\n",
            " methods\n",
            " such\n",
            " as\n",
            " using\n",
            " prompting\n",
            " techniques\n",
            ",\n",
            " task\n",
            "-specific\n",
            " instructions\n",
            ",\n",
            " or\n",
            " human\n",
            " inputs\n",
            ".\n",
            " Another\n",
            " approach\n",
            " involves\n",
            " outsourcing\n",
            " the\n",
            " planning\n",
            " step\n",
            " to\n",
            " an\n",
            " external\n",
            " classical\n",
            " planner\n",
            ".\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "\u001b[32m\"answer\"\u001b[39m"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const output = {};\n",
        "let currentKey: string | null = null;\n",
        "\n",
        "for await (const chunk of await ragChainWithSource.stream(\"What is task decomposition?\")) {\n",
        "  for (const key of Object.keys(chunk)) {\n",
        "    if (output[key] === undefined) {\n",
        "      output[key] = chunk[key];\n",
        "    } else {\n",
        "      output[key] += chunk[key];\n",
        "    }\n",
        "\n",
        "    if (key !== currentKey) {\n",
        "      console.log(`\\n\\n${key}: ${JSON.stringify(chunk[key])}`);\n",
        "    } else {\n",
        "      console.log(chunk[key]);\n",
        "    }\n",
        "    currentKey = key;\n",
        "  }\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "You've now learned how to stream responses from a QA chain.\n",
        "\n",
        "Next, check out some of the other how-to guides around RAG, such as [how to add chat history](/docs/how_to/qa_chat_history_how_to)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Deno",
      "language": "typescript",
      "name": "deno"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",
      "pygments_lexer": "typescript",
      "version": "5.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}
